Testing of distributed systems

ABSTRACT

In an embodiment, a method is provided for tracking a test. In this method, a test session identifier is transmitted to a test system. The test session identifier identifies a particular test session. A test of a component is triggered at the test system, and this test provides test results, which are received from the test system. The test results include the test session identifier, which allows the tests to be associated with the particular test session.

FIELD

The present disclosure relates generally to tests. In an embodiment, thedisclosure relates to testing of distributed systems.

BACKGROUND

Many applications are based in a distributed environment where anassignment of tasks to multiple software components (and hardwarecomponents) enables, for example, creation of business applicationsthrough composition. An example of a composed business applicationincludes business processes and composite applications based onenterprise compound services, process components, and user-interfaceapplications using core services.

Tests may be executed on such a distributed environment to verify thateach component works as expected concerning, for example, functionalcorrectness and performance. To test a particular component, mostexisting test techniques involve the insertion of specialized test codeinto production code. However, such test techniques can be inefficientbecause, for example, every new component requires the creation of newtest code. Furthermore, it can be extremely difficult to coordinatebetween all the different test codes within the distributed environment.

BRIEF DESCRIPTION OF DRAWINGS

The present disclosure is illustrated by way of example and notlimitation in the figures of the accompanying drawings, in which likereferences indicate similar elements and in which:

FIG. 1 depicts a block diagram of a distributed test environment, inaccordance with an illustrative embodiment;

FIG. 2 depicts a block diagram of a master test module, in accordancewith an embodiment, included in a computing device;

FIG. 3 depicts a flow diagram of a general overview of a method,consistent with an embodiment, for tracking tests executed at one ormore test systems;

FIG. 4 depicts a block diagram of a test environment, in accordance withanother embodiment, illustrating the tracking of tests executed at atest system;

FIG. 5 depicts a timing diagram illustrating the tracking of tests at atest system, in accordance with an embodiment;

FIG. 6 depicts a flow diagram of a detailed method, in accordance withan alternate embodiment, for tracking tests executed at multiple,distributed test systems;

FIG. 7 depicts a block diagram of a distributed test environment, inaccordance with another embodiment, illustrating a test sequence at onetest system triggering a child test sequence at another test system;

FIG. 8 depicts a timing diagram, consistent with an embodiment,illustrating the tracking of tests executed at multiple, distributedtest systems; and

FIG. 9 depicts a block diagram of a machine in the example form of acomputing device within which may be executed a set of instructions forcausing the machine to perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION

The description that follows includes illustrative systems, methods,techniques, instruction sequences, and computing machine programproducts that embody illustrative embodiments of the present invention.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide an understanding ofvarious embodiments of the inventive subject matter. It will be evident,however, to those skilled in the art that embodiments of the inventivesubject matter may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures andtechniques have not been shown in detail.

The embodiments described herein provide techniques for tracking testsof software and hardware components. In general, a central system (or amaster system) is provided to communicate with one or more remote testsystems in a test environment. This central system is configured to, forexample, manage, track, verify, and/or analyze tests executed at thetest systems. In a particular embodiment, each test or group of testsmay be tracked by the assignment of identifiers to each test or group oftests. As will be explained in more detail below, the central systemgenerates these identifiers and forwards them to the remote test systemsfor use in tracking the tests.

FIG. 1 depicts a block diagram of a distributed test environment 100, inaccordance with an illustrative embodiment. The distributed testenvironment 100 includes multiple test systems 110-112 that are incommunication with a master system 120. Generally, a “test system,” suchas test system 110, 111, or 112, is an assemblage of one or morecomponents 130-135 for testing. Each test system 110, 111, or 112 (andthe master system 120) may be hosted on one or more computing devices.For example, each test system 110, 111, or 112 may be hosted on adifferent computing device. Alternatively, the test systems 110 and 111may be hosted on one computing device while the test system 112 may behosted on a different computing device. In another example, all the testsystems 110, 111, and 112 (and even the master system 120) may be hostedon a single computing device.

As used herein, a “component,” such as component 130, 131, 132, 133,134, or 135, refers to a part or element of one or more softwareapplication or hardware apparatus. A software component can, forexample, be an element of a software system programmed to provide apredefined service or event. Examples of a software component include afunction, a subroutine, an application, and an object. As an example,the components 130-135 can be a number of components in a distributedsoftware system, such as a distributed enterprise application, orservices (e.g., an enterprise resource planning software and enterpriseperformance management software). The components 130-135 in such adistributed enterprise application may, for example, track context andapply account relationships, data, and/or rule-based constraints inbusiness applications. On the other hand, a hardware component can, forexample, be a variety of elements of a hardware system that has thecapability to process test session identifiers and test sequenceidentifiers, which are described in more detail below. Examples of sucha hardware component include dedicated circuitry or logic that ispermanently configured (e.g., a processor and a field-programmable gatearray (FPGA)) or an application-specific integrated circuit (ASIC)) toperform certain operations. It should further be noted that combinationsof software and hardware components may also be tested together. Forexample, components located at different layers (e.g., software layer,component layer, and chip layer) may be tested together. An example ofsuch a test include an integration test of a graphics card, where thetest includes calls to the graphics chip and also calls between thecomponents on the graphics chip.

Still referring to FIG. 1, the master system 120 generally is a consolethat is in communication with all the test systems 110, 111, and 112.The master system 120 can manage, track, verify, and/or analyze testsexecuted at the test systems 110-112. For example, the master system 120can control the execution of different tests at one or more of the testsystems 110-112. Furthermore, the master system 120 can also track andverify every test being executed at the test systems 110-112. Asexplained in more detail below, the master system 120 may track thetests by, for example, assigning a unique identifier to a single test ora session of tests.

FIG. 2 depicts a block diagram of a master test module 203, inaccordance with an embodiment, included in a computing device 200. Itshould be appreciated that the computing device 200 may be deployed inthe form of, for example, a personal computer, a laptop computer, aserver computer, a tablet personal computer, a personal digitalassistant, or other computing devices. The computing device 200 may beincluded in a test environment. For example, the computing device 200may form a part of the master system 120 depicted in FIG. 1. Referringto FIG. 2, in various embodiments, the computing device 200 may be usedto implement computer programs, logic, applications, methods, processes,or software to manage, track, verify, and/or analyze tests at remotetest systems, as described in more detail below.

In the example depicted in FIG. 2, the computing device 200 executes anoperating system 202 that manages the other software processes and/orservices executing on the computing device 200. These software processesand/or other services include a master test module 203, which may becomposed of an identifier distribution module 204, a test trigger module206, and a test result collection module 208. The identifierdistribution module 204 is configured to distribute or transmit variousidentifiers used in the tracking of tests. For example, the identifierdistribution module 204 is configured to distribute test sessionidentifiers, which is explained in more detail below, to all the testsystems that are in communication with the master test module 203. Incontrast, the test trigger module 206 is configured to control theexecution of tests. For example, as explained in more detail below, thetest trigger module 206 can trigger specific tests at one or more testsystems.

After the tests have been triggered, the test result collection module208 collects or receives test results from one or more test systems. Thetest results may be stored in the master test module 203 or at anotherlocation. In addition to collecting test results, the test resultcollection module 208 may also include tools that are used to verify andanalyze the test results, the tools of which are explained in moredetail below. It should be appreciated that in other embodiments, themaster test module 203 may include fewer, more, or different modulesapart from those shown in FIG. 2. For example, in an alternateembodiment, the master test module 203 may additionally include ananalysis module (not shown) that is configured to analyze and/oridentify test results associated with a particular test session.

FIG. 3 depicts a flow diagram of a general overview of a method 300,consistent with an embodiment, for tracking tests executed at one ormore test systems. The method 300 may be implemented by the master testmodule 203 and employed in the computing device 200 depicted in FIG. 2.As depicted in FIG. 3, the master system transmits a test sessionidentifier to a test system at 302. As used herein, a “test sessionidentifier” refers to a value (numeric and/or textual) that uniquelyidentifies a particular test session, which refers to a technicalbracket for each run of a set or group of tests. As an example, a testsession identifier can be a globally unique identifier, which is aspecial type of identifier that provides a unique reference number inany context. It should be noted that in other embodiments, the mastersystem may also transmit more than one test session identifier to morethan one test system, and such operations are described in more detailbelow.

With the test session identifier transmitted to the test system, themaster system at 304 thereafter triggers one or more tests of componentsat the test system. The master system may trigger the tests by, forexample, calling an execution of a specific test program or test scriptat the test system. In a particular example, the master system can calla particular test program by identifying a report name or name of aparticular class.

Once the tests of the components are triggered at the test system, thetests may generate or create various test results. A “test result,” asused herein, refers to any suitable output resulting from a test of oneor more components. For example, a component itself may generate orprovide test results. In another example, a testing program that teststhe components may generate the test results. An example of a testresult is a response time of a component to a particular input. Inanother example, a test result may include a data access time of acomponent. Such response time and data access time may be used, forexample, in analyzing the performance of the components. Other examplesof test results include data generated by the components, operationsresulting from the components, signals generated by the components, andother test results.

Still referring to FIG. 3, the master system receives the test resultsfrom the test system at 306. The master system may transmit requests fora particular test result to the test system. Alternatively, the testsystem may be automatically configured to transmit the test results tothe master system. In an embodiment, the test results include the testsession identifier, which allows the test results to be associated witha particular test session. As a result, for example, test resultsassociated with a particular test session may be identified. That is,the pairing of a test session identifier with associated test resultsallows, for example, a user or the master system itself to be able todifferentiate test results associated with a particular test sessionfrom test results of other test sessions. Such a pairing thereby, forexample, allows particular test results to be tracked, verified, and/oranalyzed on a per test session basis, which may result in fast isolationof defects in one or more components at the test system.

FIG. 4 depicts a block diagram of a test environment 400, in accordancewith one embodiment, illustrating the tracking of tests executed at atest system 110. The test environment 400 includes a master system 120in communication with a single test system 110. In this embodiment, thetest system 110 includes components 130-131, a test controller module402, and a test result collection module 403. In software, it should benoted that within the test system 110, the components 130-131, in oneexample, are independent of the test controller module 402 and the testresult collection module 403. As a result, the components 130-131 arenot customized or particularly programmed to interface with the testcontroller module 402 and the test result collection module 403. Rather,the components 130-131 can be the same components 130-131 that are usedin a production system, which may be configured to interface with otherdependent components (not shown).

The test controller module 402 generally is configured for retrievingsettings and test context 450 used in the execution of tests. As usedherein, a “test context,” such as test context 450, refers toinformation and/or data defining or specifying a behavior of one or morecomponents 130-131 during an execution of a test. As an example, thetest context 450 may include configurations and commands for use intesting a component 130 or 131. The test context 450, for example,allows the distribution of a single set of test context to differenttest systems, such as test system 110, thereby creating a single testscenario. Furthermore, in another example, the master system 120 cansimultaneously change the test scenario at different test systems bytransmitting the test context 450 to these test systems. It should benoted that different components 130-131 may recognize or interface withdifferent languages. The statements defined or described in the testcontext 450 may be in a specific language usable with a particularcomponent 130 or 131. However, in another embodiment, the statements maybe in a generic format, which is independent of the languages used bythe components 130-131. An example of such a format is Extensible MarkupLanguage (XML), and the following Table A provides an example of thetest context 450 formatted in XML. Other examples of generic formats mayinclude document formats (e.g., text file format (or .txt format)),graphic file formats (e.g., JPEG), video file formats (e.g., MPEG-4),and other formats. The components 130-131 can be configured to recognizethe generic format of the test context 450 or configured to translatethe statements in generic format into a different language.

TABLE A <?xml version=“1.0” encoding=“ISO-8859-1”?> <TEST_CASE> <TEST_SESSION>4711</TEST_SESSION> <TEST_NUMBER>17</TEST_NUMBER><BO_DATA_TO_RETURN>   <SALESORDER>   <ROOT>   <ID>12345</ID>   <ITEM>    <ID>10</ID>     <PRODUCT-ID>HT-1000</PRODUCT-ID>    <QUANTITY-CONTEXT>1</QUANTITY-CONTEXT>   </ITEM>   <ITEM>    <ID>20</ID>     <PRODUCT-ID>HT-1001</PRODUCT-ID>    <QUANTITY-CONTEXT>2</QUANTITY-CONTEXT>   </ITEM>   </ROOT>  </SALESORDER> </BO_DATA_TO_RETURN> </TEST_CASE>

It should be appreciated that the test environment 400 can include avariety of different tests executed at the test system 110. For example,the test environment 400 may be configured to test accuracies of mailingaddresses stored in a database. In this example, the master system 120can simulate an error by loading a particular test context 450 thatspecifies to the components 130-131 to access the database where theaddresses are stored, but not to search for the correct address when arequest to retrieve a particular user's address is received. That is,the test context 450 can instruct the components 130-131 to provideincorrect addresses of users in response to requests for addresses. Thistest context 450 forces an error because the components 130-131 willaccess the database of addresses, but the addresses retrieved by thecomponents 130-131 are incorrect or erroneous.

As depicted in FIG. 4, the master system 120 initially transmits thetest context 450 to the test system 110. In this embodiment, the testsession identifier may be included in the test context 450. However, inanother embodiment, the test session identifier may be transmittedseparately from the test context 450. The test controller module 402receives the test context 450, which, in this example, includes the testsession identifier. In effect, the test controller module 402 collectsthe test context 450 and stores the test context in, for example, astatic class. Whenever the components 130-131 need information about thecurrent test context 450, the components 130-131 may retrieve the testcontext 450 from the test controller module 402. Alternatively, the testcontroller module 402 may automatically transmit the test context 450 tothe components 130-131 upon receipt of the test context 450 from themaster system 120.

After the transmission of the test session identifier, the master system120 then triggers tests of the components 130-131 at the test system110. The execution of such tests based on the test context 450 resultsin the creation of test results 453 by, for example, the components130-131. The test result collection module 403 is configured to receiveor collect test results 453 created from the execution of the tests. Thetest result collection module 403 receives or collects test results 453that are created by the components 130-131. In the example of FIG. 4,the components 130-131 generate the test results 453 and transmit themto the test result collection module 403. After the tests are completed,the test result collection module 403 transmits the collected testresults 453 to the master system 120. Here, the test results 453 includethe test session identifier, and the master system 120 or a user may usethe test session identifier to track the tests by, for example,identifying the received test results 453 to be associated with aparticular test session. It should be appreciated that the test system110 may also be able to process or analyze the collected test results453. As an example, the test result collection module 403 can compressthe collected test results 453 for transmission to the master system120.

FIG. 5 depicts a timing diagram illustrating the tracking of tests at atest system, in accordance with an embodiment. In this example, thetiming diagram 500 shows the interactions between a master system 120and a test controller module 402, a component 130, and/or a test resultcollection module 403 at a test system. Initially, at 550, the mastersystem 120 transmits to the test controller module 402 a test contextthat includes a test session identifier. After transmitting the testcontext and the test session identifier, the master system 120 triggersa test at 554 of at least one component 130 at the test system.

If information regarding the current test context is needed, thecomponent 130 can request or retrieve the test context from the testcontroller module 402 at 556. With the test context, the test of thecomponent 130 is then executed at 560 based on the test context, andduring the test or after the test is completed, the component 130transmits test results at 562 to the test result collection module 403.

As depicted in FIG. 5, in one embodiment, the master system 120 maytransmit a request for the test results at 568 to the test resultcollection module 403. In another embodiment, the test result collectionmodule 403 may automatically be configured to transmit the test resultsto the master system 120. In the embodiment where the master system 120makes the request, the request may be for test results associated with aparticular test session. Here, the request identifies the particulartest session with its test session identifier. Upon receipt of therequest with the test session identifier, the test result collectionmodule 403 can identify or locate all the test results that areassociated with the test session identifier because each test result ispaired with a test session identifier. The test result collection module403 then transmits to the master system 120 only the portion of all thecollected test results that is associated with the particular testsession (or test session identifier) as requested by the master system120.

FIG. 6 depicts a flow diagram of a detailed method 600, in accordancewith an alternate embodiment, for tracking tests executed at multiple,distributed test systems. The method 600 in this example may also beimplemented by the master test module 203 and employed in the computingdevice 200 depicted in FIG. 2. As depicted in FIG. 6, a master system at602 distributes (or transmits) a particular test session identifier tomultiple test systems, each of which is hosted on different computingdevices. The master system then triggers a test sequence in one of thetest systems at 604. As used herein, a “test sequence” refers to one ormore tests that are executed within a single test session. That is, atest sequence identifies a grouping of tests within a single testsession. The tests can be associated with or assigned to a test sequencebased on a variety of relationships. It should be appreciated that atest sequence (or a test itself) can trigger other test sequences on thesame test system or on other test systems. For example, a test programtesting one component may also request a service from another componentthat is also to be tested. The relationships between the triggered testsequences can be a hierarchical relationship or parent-childrelationship where a test sequence that triggers another test sequenceis a “parent” test sequence, and the triggered test sequence is a“child” test sequence.

After the master system triggers a test sequence at one of the testsystems, the master system may then transmit requests to all the testsystems for test results at 606. Depending on whether the parent testsequence triggers any child test sequence, the master system at 608 mayreceive test results from one or multiple test systems. For example, ifthe parent test sequence does not trigger any child test sequence, thenthe master system receives test results from only the one test systemthat executed the parent test sequence. However, if the parent testsequence does trigger other child test sequences at other test systems,then the master system may receive test results from these other testsystems.

In the example described in FIG. 6, it should be noted that a testsequence identifier may also be assigned to each test sequence. As usedherein, a “test sequence identifier,” refers to a value (numeric and/ortextual) that uniquely identifies a particular test sequence. The use oftest sequence identifiers allow more granularity in tracking the testsby further enabling particular test sequences within a test session tobe identified and tracked. In an embodiment, a master system may requestparticular test results associated with a particular test sequence fromthe test systems by identifying its test session identifier in a requestto the test systems. Upon receipt of the request with the test sessionidentifier, a test result collection module can identify or locate allthe test results that are associated with the test sequence identifierbecause each test result may also be paired with a test sequenceidentifier. In addition to the test sequence identifier, the testsystems may further include the test session identifier in the testresults transmitted to the master system.

FIG. 7 depicts a block diagram of a distributed test environment 700, inaccordance with another embodiment, illustrating a test sequence at onetest system 110 triggering a child test sequence at another test system111. The test environment 700 includes a master system 120 incommunication with two test systems 110 and 111. The test system 110includes a test controller module 402, components 130-131, and a testresult collection module 403. Similarly, the test system 111 includes atest controller module 702, components 132-135, and a test resultcollection module 703.

Again, it should be noted that the test environment 400 can include avariety of different tests executed at the test systems 110 and 111. Forexample, the test system 110 can be an order procurement system that isconfigured to process orders and the test system 111 can be a suppliersystem that receives and places the orders. The test environment 400 maybe configured to test accuracies of an order of goods, such as laptopcomputers. In this example, an order of 10 laptop computers is placed bythe order procurement system (or test system 110). In this example, themaster system 120 can simulate an error by loading a particular testcontext 450 that specifies an order of 10 laptop computers is inputtedinto or received by the order procurement system. As a result, the orderprocurement system transmits an order of 10 laptop computers to thesupplier system (or test system 111). To simulate an error, the testcontext 450 also specifies to the components 132-135 of the suppliersystem to provide an inaccurate confirmation of the order from the orderprocurement system. For example, the components 132-135 can beconfigured to transmit a confirmation of 8 laptop computers rather thanthe original order of 10 laptop computers.

In the example of FIG. 7, the master system 120 initially distributestest contexts 450 to both test systems 110 and 111. The test contexts450 include multiple test contexts associated with different testsessions, with each test context paired with or assigned a test sessionidentifier. The master system 120 then triggers a test sequence at testsystem 110, which executes tests of components 130-131 at the testsystem 110. This trigger may include a test session identifier thatidentifies a particular test session to be executed. If the components130-131 need the test context 450′ associated with this particular testsession, the components 130-131 can retrieve from the test controllermodule 402 the test context 450′ associated with this particularsession.

During testing, the initial (or parent) test sequence triggers 756 achild test sequence of one component 132 at the test system 111. Thiscomponent 132 at the test system 111 may retrieve the particular testcontext 450′ from its test controller module 702 based on a test sessionidentifier, which may be included in the triggering 756 by the parenttest sequence at test system 110.

In this example, both test sequences at test systems 110 and 111 may beexecuted in parallel, and may end at different times. The test results453 and 753 may not be available to the master system 120 until all thetest sequences have been completed. Before the completion of the testsequences, the master system 120 may query each test system 110 or 111for its state of testing. In one embodiment, each test system 110 or 111may track a state of its testing, which allows the master system 120,for example, to monitor the activities of a test run. Examples of statesinclude triggered, initialized, running, ended, cancelled, and otherstates. It should be noted that the “triggered” state identifies atracking of dependent test sequences, and may, for example, be used toidentify whether test sequences within a test session are still running.

The master system 120 may periodically transmit requests to the testsystems 110 and 111 for their state of testing. In turn, the test system110 may transmit a response to the master system 120 with the state ofthe test. This state allows the master system 120 to identify, forexample, when tests associated with a particular test session and/ortest sequence are completed at each test system 110 or 111. When themaster system 120 has identified that all the tests are completed, themaster system 120 may thereafter transmit requests to the test systems110 and 111 for the test results. In response, the test resultcollection modules 403 and 703 transmit their test results 453 and 753,respectively, to the master system 120.

FIG. 8 depicts a timing diagram 800, consistent with an embodiment,illustrating the tracking of tests executed at multiple, distributedtest systems. In this example, the timing diagram 800 shows theinteractions between a master system 120 and test systems 110 and 111.The test system 110 includes a test controller module 402, a component130, and a test result collection module 403. Similarly, the test system111 includes a test controller module 702, a component 132, and a testresult collection module 703.

Initially, the master system 120, as depicted at 850 and 854, transmitsa test context, which includes a test session identifier, to both testcontroller modules 402 and 702. After the test context is distributed,the master system 120 triggers a test at 856 of the component 130 at thetest system 110. In response, the component 130 retrieves the testcontext at 858 from the test controller module 402, and a test is thenexecuted at 860 based on the retrieved test context.

In this example, the initial or parent test sequence triggers at 880 achild test sequence of the component 132 at the test system 111. At thetest system 111, the component 132 retrieves the test context at 882from the test controller module 702, and another test sequence isexecuted at 886 based on the retrieved test context. After thecompletion of the test, the component 132 transmits at 888 the testresults to the test result collection module 703.

During tests of the component 132 at the test system 111, the testing ofthe other component 130 at the other test system 110 has completed. Thecomponent 130 then transmits at 862 the test results to the test resultcollection module 403 and returns back to the master system 120. Itshould be noted that the test is started by, for example, calling amethod of the component 130 synchronously, thereby returning back to themaster system 120 after the method has come to an end. However, thecomponent 132 is called asynchronously, for example, by the component130, and therefore returns to nowhere and simply stops after thecomponent 132 has done its work.

After the completion of all the tests at the test systems 110 and 111,the master system 120 may transmit requests, as depicted at 868 and 892,to both the test result collection modules 403 and 703 for the testresults of, for example, a particular test session. Upon receipt of therequest, the test result collection modules 403 and 703 retrieve thetest results associated with the particular test session based on a testsession identifier, and then transmit responses with the test results tothe master system 120.

It should be noted that test session identifiers and/or the testsequence identifiers may be used to identify particular test resultsthat are associated with a particular test session and/or test sequence.A user, for example, may use this information to identify a particulartest session or a test sequence within a test session that is working orhas failed. Alternatively, the user may use this information to apply,for example, various analyses to test results of particular testsessions or test sequences. The identification of test resultsassociated with particular test sessions and/or test sequences can bemade by a user. However, it should be noted that in an embodiment, themaster system 120 may also have some capability to isolate or identifyparticular test results. For example, still referring to FIG. 8, themaster system 120 may request from the test result collection modules403 and 703 all test results from different test sessions. In turn, thetest result collection modules 403 and 703 transfer all their testresults with their associated test session identifiers and test sequenceidentifiers. The master system 120 may store all the test results fromtest systems 110 and 111 in its non-volatile memory.

When a request is subsequently made to the master system 120 forparticular test results associated with a particular test session, themaster system 120 can compare the test session identifier associatedwith each test result with a particular test session identifierassociated with the requested test session. The master system 120 canthen identify the test results that are associated with the requestedtest session based on a match of their test session identifiers with theparticular test session identifier associated with the requested testsession.

FIG. 9 depicts a block diagram of a machine in the example form of acomputing device 200 within which may be executed a set of instructionsfor causing the machine to perform any one or more of the methodologiesdiscussed herein. In alternative embodiments, the machine operates as astandalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine may operate in thecapacity of a server or a client machine in server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. Embodiments may also, for example, be deployed bySoftware-as-a-Service (SaaS), Application Service Provider (ASP), orutility computing providers, in addition to being sold or licensed viatraditional channels.

The machine is capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example of the computing device 200 includes a processor 902 (e.g.,a central processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 904 (e.g., random access memory (a type of volatilememory)), and static memory 906 (e.g., static random access memory (atype of volatile memory)), which communicate with each other via bus908. The computing device 200 may further include video display unit 910(e.g., a plasma display, a liquid crystal display (LCD) or a cathode raytube (CRT)). The computing device 200 also includes an alphanumericinput device 912 (e.g., a keyboard), a user interface (UI) navigationdevice 914 (e.g., a mouse), a disk drive unit 916, a signal generationdevice 918 (e.g., a speaker), and a network interface device 920.

The disk drive unit 916 (a type of non-volatile memory storage) includesa machine-readable medium 922 on which is stored one or more sets ofdata structures and instructions 924 (e.g., software) embodying orutilized by any one or more of the methodologies or functions describedherein. The data structures and instructions 924 may also reside,completely or at least partially, within the main memory 904 and/orwithin the processor 902 during execution thereof by computing device200, with the main memory 904 and processor 902 also constitutingmachine-readable, tangible media.

The data structures and instructions 924 may further be transmitted orreceived over a computer network 950 via network interface device 920utilizing any one of a number of well-known transfer protocols (e.g.,HyperText Transfer Protocol (HTTP)).

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., the computing device 200) or one or morehardware modules of a computer system (e.g., a processor 902 or a groupof processors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor 902 or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor 902 configured using software, thegeneral-purpose processor 902 may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor 902, for example, to constitute a particular hardware moduleat one instance of time and to constitute a different hardware module ata different instance of time.

Modules can provide information to, and receive information from, otherhardware modules. For example, the described hardware modules may beregarded as being communicatively coupled. Where multiples of suchhardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)that connect the hardware modules. In embodiments in which multiplehardware modules are configured or instantiated at different times,communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation, and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors 902 that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors 902 may constitute processor-implementedmodules that operate to perform one or more operations or functions. Themodules referred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors 902 orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors 902, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors 902may be located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors 902 may be distributed across a number of locations.

While the embodiment(s) is (are) described with reference to variousimplementations and exploitations, it will be understood that theseembodiments are illustrative and that the scope of the embodiment(s) isnot limited to them. In general, techniques for tracking tests may beimplemented with facilities consistent with any hardware system orhardware systems defined herein. Many variations, modifications,additions, and improvements are possible.

Plural instances may be provided for components, operations orstructures described herein as a single instance. Finally, boundariesbetween various components, operations, and data stores are somewhatarbitrary, and particular operations are illustrated in the context ofspecific illustrative configurations. Other allocations of functionalityare envisioned and may fall within the scope of the embodiment(s). Ingeneral, structures and functionality presented as separate componentsin the exemplary configurations may be implemented as a combinedstructure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements fall within the scope of the embodiment(s).

1. A computer-implemented method of testing at a plurality of testsystems, the method comprising: distributing a particular test sessionidentifier to the plurality of test systems, the plurality of testsystems including a first test system and a second test system, theparticular test session identifier identifying a particular testsession; triggering a first test sequence in the first test system, thefirst test sequence configured to trigger a second test sequence in thesecond test system, the first test sequence further configured toprovide a first test result at the first test system, the second testsequence configured to provide a second test result at the second testsystem; transmitting a plurality of requests to the plurality of testsystems for a test result; and receiving the first test result and thesecond test result from the plurality of test systems in response to theplurality of requests, the first test result and the second test resultincluding the particular test session identifier that allows the firsttest result and the second test result to be associated with theparticular test session.
 2. The computer-implemented method of claim 1,wherein the particular test session includes at least one test sequence,the at least one test sequence including a test of a component includedin at least one of the plurality of test systems.
 3. Thecomputer-implemented method of claim 1, further comprising transmittinga test context to the plurality of test systems, the testing being basedon the test context.
 4. The computer-implemented method of claim 1,further comprising storing the first test result and the second testresult in non-volatile memory.
 5. The computer-implemented method ofclaim 1, further comprising: transmitting the plurality of requests tothe plurality test systems for a state of the testing at each of theplurality of test systems; and receiving the state of the testing fromthe each of the plurality of test systems, the state of the testingallowing identification of a completion of the testing.
 6. Amachine-readable medium that stores instructions, which, when performedby a machine, cause the machine to perform operations comprising:distributing a particular test session identifier to a plurality of testsystems, the plurality of test systems including a first test system anda second test system, the particular test session identifier identifyinga particular test session; triggering a first test sequence in the firsttest system, the first test sequence configured to trigger a second testsequence in the second test system, the first test sequence furtherconfigured to provide a first test result at the first test system, thesecond test sequence configured to provide a second test result at thesecond test system; transmitting a request to the plurality of testsystems for a plurality of test results; receiving the plurality of testresults from the plurality of test systems, the plurality of testresults including the first test result and the second test result, theplurality of test results including a test session identifier; comparingthe test session identifier with the particular test session identifier;and identifying the first test result and the second test result beingassociated with the particular test session based on the test sessionidentifier included in the first test result and the second test resultmatching the particular test session identifier.
 7. The machine-readablemedium of claim 6, wherein the plurality of test results furtherincludes a plurality of test sequence identifiers, each test sequenceidentifier uniquely identifying a test sequence within a single testsession.
 8. The machine-readable medium of claim 6, wherein the firsttest system includes a software component, the first test sequence beinga plurality of tests on the software component.
 9. The machine-readablemedium of claim 8, further comprising transmitting a test context to thefirst test system, the software component being tested based on the textcontext.
 10. The machine-readable medium of claim 6, wherein theparticular test session identifier is a globally unique identifier. 11.The machine-readable medium of claim 6, wherein the first test system ishosted on a computing device and the second test system is hosted on adifferent computing device.
 12. A computing device comprising: at leastone processor; and a memory in communication with the at least oneprocessor, the memory being configured to store an identifierdistribution module, a test trigger module, and a test collection modulethat are executable by the at least one processor, the identifierdistribution module having instructions that when executed by the atleast one processor, cause operations to be performed, the operationscomprising transmitting a test session identifier to a test system, thetest session identifier identifying a particular test session, the testtrigger module having instructions that when executed by the at leastone processor, cause operations to be performed, the operationscomprising triggering a test of a component at the test system, the testof the component providing a plurality of test results, and the testresult collection module having instructions that when executed by theat least one processor, cause operations to be performed, the operationscomprising receiving the plurality of test results from the test system,the plurality of test results including the test session identifier thatallows the plurality of test results to be associated with theparticular test session.
 13. The computing device of claim 12, whereinthe test system includes a test result collector module that isconfigured to receive the plurality of test results from the componentat the test system and transmit the plurality of test results to thetest collection module.
 14. The computing device of claim 12, whereinthe identifier distribution module has instructions that when executedby the at least one processor, cause operations to be performed, furthercomprising transmitting a test context to the test system, the componentbeing tested based on the test context.
 15. The computing device ofclaim 14, wherein the test system includes a test controller module thatis configured to receive the test context and to distribute the testcontext to the component at the test system.
 16. The computing device ofclaim 14, wherein the test context is formatted in Extensible MarkupLanguage (XML).
 17. The computing device of claim 12, wherein the testis associated with a particular test sequence in a plurality ofassociated test sequences, the test session identifier furtheridentifying a grouping of the plurality of associated test sequences.18. The computing device of claim 12, wherein the test system is hostedon a different computing device that is in communication with thecomputing device.
 19. The computing device of claim 12, wherein the testresult collection module has instructions that when executed by the atleast one processor, cause operations to be performed, furthercomprising: transmitting a request to the test system for a state of thetest; and receiving the state of the test from the test system, thestate of the test allowing identification of a completion of the test.20. The computing device of claim 12, wherein the component is one of aplurality of components in a distributed system.